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ABSTRACT 

Current  activities  on  the  project  are  reviewed  under  the 
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1)  Segmentation 

2)  Local  feature  detection 
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1. 


1 .  introduction 

This  project  is  concerned  with  the  study  of  advanced 
techniques  for  the  analysis  of  reconnaissance  imagery..  It 
is  being  conducted  under  Contract  DAAG-53-76-C-0138  (JJARPA 
Order  3206),  monitored  by  the  U.S.  Army  Night  Vision  and 
Electro-Optics  Laboratory,  Ft.  Belvoir,  VA  (Dr ./George  Jones) . 
The  Westinghouse  Systems  Development  Divisiofi^  under  a  sub¬ 
contract,  is  collaborating  on  implementation  and  application 
aspects . 

The  previous  phase  of  the  project,  entitled  "Image  Under¬ 
standing  Using  Overlays",  was  concluded  during  the  past  report¬ 
ing  period.  Accomplishments  under  this  phase  are  summarized 
in  a  Final  Rejport  dated  May  1980  [1]  ,  which  also  contains  a 
bibliography  of  all  reports  and  papers  produced  during  this 


period^ 

The  current  phase  of  the  project  is  concerned  with  three 
principal  areas:  (a)  comparative  analysis  of  segmentation  tech¬ 
niques  applied  to  FLIR  imagery;  (b)  development  of  an  inference- 
based  approach  to  target  detection  on  FLIR  imagery;  and  (c) 
optical  flow  analysis  of  time-varying  imagery.  Work  in  area  (b) 
is  in  progress  and  will  be  described  in  fortlvSoming  technical 
reports.  Area  (a)  has  emphasized  methods  based  on  hierarchical 
("pyramid")  image  representations,  some  of  which  are  reviewed 
in  this  report  and  in  individual  Technical  Reports  [2,3].  Other 
individual  reports  [4,5]  deal  with  some  of  the  work  done  in  area 
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*  (c) .  In  addition,  the  project  is  preparing  software  contri- 

|  butions  to  the  DARPA/DMA  Image  Understanding  Testbed;  the 

first  of  these  will  be  a  general-purpose  software  package  for 
implementing  relaxation  processes  at  the  pixel  level.  i 

j 

This  report  reviews  activities  on  the  project  during  the  ■ 

.1 

j  period  April  1980-January  1981.  This  work  is  covered  under 

!  i 

)  the  headings  of  segmentation;  local  feature  detection;  feature 

i 

'  linking;  and  hierarchical  representation.  The  work  is  summarized  ' 

I  i 

’  only  briefly,  since  it  is  covered  in  greater  detail  in  individual  j 

1  | 

technical  reports  and  Image  Understanding  Workshop  papers. 


.  Segmentation 


2.1  Color  pixel  classification 

When  pixels  in  a  black-and-white  image  are  classified 
by  thresholding  their  gray  levels,  gradient  magnitude  infor¬ 
mation  can  be  used  in  various  ways  as  an  aid  in  threshold 
selection.  In  particular,  a  histogram  of  the  gray  levels  of 
pixels  whose  gradient  magnitudes  are  low  has  sharper  peaks  and 
deeper  valleys  than  the  histogram  of  the  entire  image,  since 
the  low-gradient  pixels  tend  to  come  from  the  interiors  of 
regions,  not  from  region  border  zones;  it  is  easier  to  choose 
useful  thresholds  (at  valley  bottoms)  from  this  improved  histo¬ 
gram.  Analogously,  when  pixels  in  a  color  or  multispectral 
image  are  classified  on  the  basis  of  their  spectral  signatures, 
the  color  gradient  magnitude  can  be  used  as  an  aid  in  defining 
decision  surfaces  that  separate  clusters  of  pixels  having  like 
signatures.  In  fact,  a  scatterplot  of  the  signatures  of  pixels 
whose  color  gradient  magnitudes  are  low  has  more  clearly  separ¬ 
ated  clusters  than  the  scatter  plot  of  the  entire  image,  for  the 
same  reason  as  in  the  grayscale  case.  This  phenomenon  is  illus¬ 
trated  in  Figure  1.  For  further  details  and  additional  examples, 
see  [6]  . 

2 . 2  Mosaicking 

When  aerial  photographs  are  combined  into  a  photomosaic,  seams 
are  often  apparent  between  the  parts.  These  seams  are  caused  by 
gray  level  differences  due  to  the  different  conditions  under 
which  the  parts  were  recorded.  A  relaxation  method  has  been 


developed  that  generates  a  gray  level  correction  function  such 
that,  when  this  function  is  subtracted  from  the  mosaic,  the 
seams  are  eliminated,  but  the  details  of  the  photographs  are 
not  affected.  The  algorithm  does  not  assume  any  specific  types 
of  gray  level  differences  among  the  parts,  nor  does  it  require 
the  existence  of  overlaps  between  the  parts ,  and  it  can  be  used 
for  arbitrary  numbers  of  parts;  but  it  does  have  the  drawback 
that  if  a  seam  coincides  with  an  edge  between  two  regions,  that 
edge  will  be  eliminated.  The  algorithm  constructs  a  seam- 
eliminating  function  which,  when  subtracted  from  the  mosaic, 
causes  the  gray  levels  at  pairs  of  adjacent  points  on  opposite 
sides  of  a  seam  to  become  equal,  and  which  otherwise  is  as 
smooth  as  possible.  An  example  of  mosaic  seam  elimination 
using  this  algorithm  is  shown  in  Figure  2.  Other  examples, 
and  further  details,  can  be  found  in  [7], 


I .  Local  Feature  Detection 

3 . 1  Higher-order  edge  detectors 

One  way  to  define  edge  detectors  for  digital  images  is 
to  fit  a  polynomial  surface  to  a  neighborhood  of  each  pixel, 
and  take  the  magnitude  of  the  gradient  of  that  surface  as  an 
estimate  of  edgeness.  The  polynomial  fitting  process  is 
usually  carried  out  for  symmetric  neighborhoods,  using  poly¬ 
nomials  of  degree  1  or  2.  Using  least  squares  fitting  by 
orthonormalization  and  Richardson  extrapolation,  one  can 
calculate  such  edge  estimates  for  other  classes  of  neighbor¬ 
hoods,  and  for  higher-order  polynomial  models  [8] . 

As  further  application  of  this  approach,  edge  detectors 
can  be  defined  based  on  least  squares  surface  fitting  in 
which  the  surface  is  a  step  edge  superimposed  on  a  low-order 
polynomial  function.  This  makes  it  possible  to  "filter"  op¬ 
timal  step-based  operator  responses  so  as  to  discriminate 
against  noise  responses,  by  rejecting  responses  for  which  the 
fit  is  poor,  without  discriminating  against  low-contrast  edge 
(which  is  unavoidable  if  thresholding  is  used  for  noise  sup¬ 
pression)  .  An  example  of  such  edge  "filtering"  is  shown  in 
Figure  3.  For  other  examples,  and  further  details,  see  [9]. 

3 . 2  Edge  evaluation 

A  method  of  evaluating  edge  detector  output  has  been 


developed,  based  on  the  local  good  form  of  the  detected  edges 
It  combines  two  desirable  qualities  of  well-formed  edges  — 
good  continuation  and  thinness.  The  measure  has  the  expected 


behavior  for  known  input  edges  as  a  function  of  their  blur 
and  noise.  It  yields  results  generally  similar  to  those 
obtained  with  measures  based  on  discrepancy  of  the  detected 
edges  from  their  known  ideal  positions,  but  it  has  the  ad¬ 
vantage  of  not  requiring  ideal  positions  to  be  known.  It 
can  be  used  as  an  aid  to  threshold  selection  in  edge  detection 
(pick  the  threshold  that  maximizes  the  measure) ,  as  a  basis 
for  comparing  the  performances  of  different  detectors,  and  as 
a  measure  of  the  effectiveness  of  various  types  of  prepro¬ 
cessing  operations  facilitating  edge  detection.  This  method 
is  described  in  detail  in  a  separate  Technical  Report,  where 
examples  of  its  performance  are  also  given  [10] . 
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4 .  Feature  Linking 

4 . 1  Edge  segment  linking 

A  system  of  programs  that  links  edge  segments  based  on 
both  gray  level  and  geometric  criteria  has  been  developed 
and  applied  to  the  detection  of  buildings  and  roads  on 
aerial  photographs.  Preliminary  results  using  these  programs 
were  described  in  [11] ;  a  more  detailed  description,  and 
numerous  additional  results,  are  presented  in  [12].  Further 
work  along  these  lines  led  to  the  development  of  figures  of 
merit  for  linking  compatible  segments  (i.e.,  segments  that 

could  be  consecutive  sides  of  an  object)  and  antiparallel  seg¬ 
ments  (i.e.,  segments  that  could  be  opposite  sides).  For  compa¬ 
tible  pairs,  the  figure  of  merit  is  based  on  the  geometrical 
configuration  of  the  segments,  the  similarity  of  the  gray  levels 
on  their  "object"  sides,  and  the  similarity  between  their  object 
sides  and  the  line  joining  their  endpoints.  For  antiparallel  pairs, 
it  is  based  on  the  homogeneity  of  gray  level  between  the  edges  and 
the  amount  of  overlap  between  them.  These  figures  of  merit 
have  highly  bimodal  histograms,  making  it  quite  easy  to  decide 
which  pairs  of  segments  should  be  linked, as  illustrated  in 
Figures  4  and  5.  They  should  be  useful  in  the  design  of 
relaxation-like  schemes  for  classifying  edge  segments.  For 
further  details,  and  many  additional  results,  see  [13,14]. 

4 . 2  Reconstruction  from  gray-weighted  medial  axes 

A  method  of  defining  a  "min-max  medial  axis  transformation" 


(MMMAT)  for  grayscale  images,  based  on  iterated  local  MIN  and 
MAX  operations,  was  described  in  a  previous  report  [1].  This 


transformation  associates  with  each  pixel  a  vector  of  gray 
level  increments,  and  exact  reconstruction  of  the  image  is 
possible  from  these  vectors.  Moreover,  good  approximations 
to  the  image  can  be  reconstructed  using  only  the  strongest 
components  of  the  strongest  few  vectors.  A  few  illustrations 
of  this  were  given  in  an  earlier  report;  further  details  and 
additional  examples  can  be  found  in  [15] . 
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5  Hierarchical  Representation 

Extensive  work  has  been  done  on  this  project  on  the  use 
of  pyramid  and  quadtree  structures  for  image  representation 
and  processing.  The  work  done  in  this  area  through  March 
1980  was  summarized  in  [16] .  In  this  section  we  briefly 
summarize  developments  in  this  area  during  the  past  reporting 
period . 

5 . 1  Quadtree-to-raster  conversion 

An  algorithm  for  converting  quadtree  representations 
of  binary  images  to  row-by-row  (e.g.,  run-length)  representa¬ 
tions  was  described  and  partially  analyzed  in  an  earlier  report. 
More  recently,  a  comparative  study  and  complete  analysis  of 
four  such  algorithms  has  been  conducted  [17] .  The  simplest 
algorithm  is  a  straightforward  top-down  approach  that  visits 
each  run  in  a  row  in  succession  starting  at  the  root  of  the 
tree;  the  other  algorithms  proceed  in  a  manner  akin  to  an  in- 
order  tree  traversal.  The  analysis  shows  under  what  circum¬ 
stances  each  algorithm  is  preferable.  They  have  all  been 
shown  to  have  execution  times  proportional  to  the  sum  of  the 
heights  of  the  blocks  comprising  the  image. 

5 . 2  Quadtree-based  image  smoothing 

Two  methods  for  smoothing  an  image  using  quadtree  approxi¬ 
mations  to  the  image  have  been  developed.  One  uses  the  sizes 
of  the  leaves  in  the  quadtree  to  determine  neighborhood  sizes 
over  which  to  apply  the  smoothing.  The  other  method  maps  each 
image  gray  level  i  into  the  gray  level  j  into  which  i  most 


frequently  maps  when  we  replace  the  level  of  each  pixel  by 
the  level  of  the  quadtree  leaf  to  which  it  belongs.  Results 
obtained  using  these  methods,  as  well  as  a  local  histogram 
peak  sharpening  method,  are  shown  in  Figure  6.  The  second 
quadtree-based  method  seems  to  give  the  best  results.  Addi¬ 
tional  examples,  and  detailed  descriptions  of  the  methods, 
can  be  found  in  [18]. 

5 . 3  Edge  pyramids  and  quadtrees 

An  edge  (or  curve)  pyramid  is  a  sequence  of  successively 
lower-resolution  versions  of  an  image,  each  containing  a 
summary  of  the  edge  information  in  its  predecessor.  This 
summary  includes  the  average  edge  magnitude  and  direction  in 
each  "block"  of  the  higher-resolution  image,  together  with 
an  intercept  in  that  block  and  a  measure  of  the  error  in  the 
direction  estimate.  An  edge  quadtree,  analogously,  is  a 
variable-resolution  representation  of  the  edge  or  curve 
information  in  the  given  image,  constructed  by  recursively 
splitting  the  image  into  quadrants  based  on  magnitude, 
direction,  intercept,  and  error  information.  Advantages  of 
these  representations  include  their  registration  with  the 
original  image,  their  ability  to  represent  many  edges  or  curves 
in  a  single  tree  structure,  and  their  ability  to  perform  many 
operations  on  the  represented  data  efficiency.  A  detailed 
description  of  these  representations,  together  with  examples, 
can  be  found  in  a  separate  Technical  Report  [2] . 


5.4  Pyramid  linkin 


When  an  image  is  smoothed  using  small  blocks  or 
neighborhoods,  the  results  may  be  somewhat  unreliable  due 
to  the  effects  of  noise  on  small  samples.  When  larger 
blocks  are  used,  the  samples  become  more  reliable,  but  they 
are  more  likely  to  be  mixed,  since  a  large  block  will  often 
not  be  contained  in  a  single  region  of  the  image.  A  compro¬ 
mise  approach  is  to  use  several  block  sizes,  representing 
versions  of  the  image  at  several  resolutions,  and  to  carry 
out  the  smoothing  by  means  of  a  cooperative  process  based 
on  links  between  blocks  of  adjacent  sizes.  These  links 
define  "block  trees"  which  segment  the  image  into  regions, 
not  necessarily  connected,  over  which  smoothing  takes  place. 

The  basic  "pyramid  linking"  scheme  was  described  in  an 
earlier  report.  Further  experiments  with  this  scheme  have 
led  to  some  improvements  over  the  original  method,  based  on 
better  ways  of  initializing  the  process  and  measuring  the 
link  merit.  A  detailed  description  of  these  experiments 

and  their  results  can  be  found  in  a  separate  Technical 
Report  [3] .  It  has  also  been  found  that  forced-choice 
linking  of  blocks  to  larger  blocks  is  not  necessary;  one 
can  use  weighted  links,  recomputing  the  weights  at  each 
iteration,  and  it  turns  out  that  the  weights  converge  to 
0's  and  l's  as  the  process  stabilizes.  Generalizations 
of  this  approach  to  image  features  other  than  gray  level, 
including  color  signatures  and  textural  properties,  have 
also  been  investigated  and  will  be  described  in  future  reports. 
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Figure  6.  Quadtree-based  image  smoothing 
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